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FOREWORD 



In !973» under a grant from the U.S. Department of Labor, the 
Bureau of Soda! Science Research undertook an assessment of the adminis- 
trative feasibility of the introduction in the Work Incentive Program 
(WIN) of a voucher system for skill training. The program was designed 
to rearrange relationships antong clients, WIN staff, and providers of 
training, by tra. sferrlng responsibility for decision-making and negotia- 
tions for training arrangements from staff to clients, and shifting 
the staff role from that of Intermediary between client and trainer to 
one of provider of Information services for clients.^ 

The original feasibility test called for issuing vouchers for 
vocational school training to a limited number of clients, and addressing 
several concrete Issues of admlni st* at i ve feasibility 

• whether program participants would find their responsibilities 
as voucher clients acceptable, or whether they would prefer 
intercession by WIN program staff with potential trainers, 

• whether clients could find suitable training within the limits 
of the program guidelines, 

• whether members of the WIN ^taff would be able to shift the 
emphasis in their work from rrtediation of the relationship 
between client and trainer to that of facilitation and provision 
of Informat Ion; and 

• whether trainers (vocational schools, and later, employers) 
would accept voucher clients as trainees without prior screening 
or intercession by WIN program staff. 



Briefly, WIN pro\ \des placement and training services to certain 
recipients of Aid to Farii 1 (es with Dependent Children (AFDC). The program 
is jointly administered by the Departments of Labor and Health, Education, 
and Welfare. At the local level, the two administrative arms are the 
Employment Service (ES) and a Separate Administrative Unit (SAU) which pro- 
vides for day care for the children of WIN participants, medical care, and 
other social services. 
2 

For details on various aspects of the project as a whole, see 
Dunning, 1976a and 1976b, Dunning and Unger. 197S; Richardson. 1977, Richardson 
and Sharp, 197^* *nd 1975. Several additional reports are currently in 
preparation and will be published during 1977- 

iv 



Following the completion of the institutional training study 
("Phase !••)» the project was simultaneously expanded to Include follow- 
up evaluation of training and labor force outcomes of the voucher recip- 
ients, and extended ({n "Phase II") to a test of vouchers for on-the-job 
training (OJT) . The grant was modified agair to provide for a test in 
1976 of a voucher program which offered both modes of training at the 
same time, to be selected as the voucher cliCnt decided ('Phase III'). 

Throughout the active field (voucher- i ssu ing) phases of each of 
the voucher programs, BSSR rr^intalned an on-site staff, charged witn 
responsibility to help to resolve ambiguities In program operational 
procedures (developed In BSSR's Washington office); to discuss with WIN 
staff members their reactions to, and ability to function within, the 
program guidelines, and to collect data on the characteristics of voucher 
prograr. and regular WIN clients and on the general setting in which the 
program was operating. One result of the on-site activity was that we 
accumulated large amounts of information on the day-to-day aspects of 
the implementation and operation of the voucher program, ano Identified 
a variety of situational factors which affected the shape which the 
program actually assumed as it went along, we also encountered a number 
of threats to the integrity of the research enterprise itself, and 
particularly to the conditions required for carrying out valid follow-up 
evaluat ions. 

In order to make more than passing and anecdotal use of this 
information, tfSSR asked Dr. Carol Greenhouse to develop a general "adminit 
trative history" of the Portland projects, with emphasis on the events 
and decisions which affected the development and implementation of the 
voucher system and on the evolution of the research on the programs. 
Th^ results of that effort arj the subject of this report. 

Greenhouse, a social anthropolog* st. came to the project in 
January, 1976, well after the vouchered OJT progranrt was under way In the 
fiel^J. Thus, the reconstruction of the history of the voucher tests 
'-equired that she start from scratch to gather information from all of 
those. Including BSSR staff, who were active participants In the attempt 
t*^ establish and assess the program. She worked with minimal guidance 
from BSSR. project staff were Interviewed for her study Just as were OOL 
and WIN staff members at national, regional, state, and local levels. 



Th« events (and their consequences) whfch are described In the 
report ere of the sort which are usually considered the "nulsdnces" of 
program Implementation and of research In real settings. It Is regularly 
assumed In program evaluation research that bothersome events of this 
type Mill have some effect on the follow-up findings, out they are rarely 
treated systematically. Greenhouse htv done just that: iin#n? 'r ip»t-d and 
uncontrollable environmental events are treated directly as date, and an 
assessment made of thel-' effects on what we know about the outcomes of the 
field tests and about voucher Ing In WIN In general. 

This is an Interesting report from several points of view. For 
example, the unusual opportunity to compile a history of the Implementation 
of the voucher system gave us valuable Insight on the programs, to supple- 
ment the general evaluation data collected In follow-up Interviews with 
clients and with the schools and OJT employers who were In one way or 
another Involved In the field tests. 

There Is another level on which these materials are valuable, 
however. The Greenhouse report raises some Important questions about 
relations between research activities and program operations In general, 
and about ways In which their respective objectives may best be meshed 
in real settings. This Is a matter of concern to those, researchers 
and program administrators alike, who seek to develop high-quality and 
useful studies of social proarams and their effects upon their clienteles. 

Finally, the analysis raises Important questions about appropriate 
strategies for program evaluat Ion research. In particular. It suggests 
that there are dangers In carrying out any evaluation of program effects 
without well-developed Information on what, If any, relationship there 
Is between the program-as- designed and the "program-as-operated." (WeIss, 
1972a, 51). 

These broader points are discussed In "A Comment on Some larger 
Issues" which follows the body of the Greenhouse report. 
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THE FEASIBILITY OF FEASIBILITY TESTING: 
OBSERVATIONS FROM THfc PORTLAND 
WIN VOUCHER TEST 



t ntroduct ton 

Far from precipitating the events that accompanied its life in 
Portland, the voucher project represented only a new link in a preexisting 
and continuing chain of personal and bureaucratic relationships. At most, 
its existence triggered specific events that might not have occurred 
otherwise, but even those reactions developed along program-related lines 
that weri? substantively unrelated to the research As one high-level 
a-'-^inistrator said, the various parties interested in BSSR's research 
all accommodated to the contract (and to each other) for the sake of 
their own goals; however, they lived In different corners of the con- 
tract and each had his own view of the Interrelationships that existed as 
a result of it He drew a square for me and said. "This is the con- 
tract," and then proceeded to draw four dots, one in each corner of the 
square. Each dot represented an administrative unit, ani the last one 
was the Bureau. "You see, we are all within the framework," he said, 
"but we all want different things from it " 

His diagram appears to have been accurate The regioial office 
felt that in agreeing to allow the project lo operate it was fulfilling 
its responsibility to spend money on research Along with some srrong 
personal biases against research ("As a taxpayer, I resent the voucher 
project"), a rather strong sense of administrative jealously makes th.' 
distribution of regional monies to the states an awkward and painful 
process for these officials The region is caught between bureaucratic 
levels: it monitors the states with the authority of the federal govern- 
ment, yet it Is not an autonop^ous unit itself Furthermore, it is such 
a small office (four then, and now three professionals) that there is 
ample opportunity for personal views to color administrative actions. 

The state administrators' interest in the research had less to 
do with the research itself than it did with one of the by products 
of the establishment of the research program, i e , money One state 
official said, "We wanted the research because we wanted something 
for nothing." Eventually, he got what he wanted, but, as we shall 
see, it took three years for that to happen 
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Generally, the local of f i ce--except for one man--did not have 
any particular Interest in research, although some of the stalT^ere 
interested in vouchenng as a concept. Rather, the local staff's 
approach to the research was guided by their hope to supplement 
regular program funds and to find some relief fra.i the program's 
lim.tations, caused by shortages in funding, and by more general 
program emphasis (especially placement). They inaccuratel> envisioned 
the voucher project as a new program that would be available to their 
clients, i.e . a new and less limited set of resources. It was not: 
voucher ing was simply a new route to the same ends. 

For its part. BSSR planned a quas i -exper imen t t..at. in addition 
to the Department of labor's substantive purposes, had technical demands 
and goals of its own. BSSR expected that the regular program would 
continue unchanged by the presence of the research, would constitute a 
baseline by which the voucher project could be evaluated, and that 
the research itself would set up a parallel, optional method or service 
del I very to cl ients 

Ultimately, tach of these goals developed an asoec*^ vf mutual 
exclusivity, i.e., one could not be achieved without weakening another. 
Thus, they came into varying degrees of conflict, and it may never be 
possible to untangle ait the threads of accommodiition and compromise 
that led to the end of the project three years later. 

Except for BSSR, the roles and goal conflicts that became an 
integral part of the voucher project are endemic to the agency's continu- 
ing decision-making process. The Bureau was an accidental spectator 
drawn into the constant interplay among levels of a public agency. In 
other words, in many cases, the voucher project simply act ivated pre- 
existing conflicts (e.g.. between SAU and ES. or between levels of the 
agency hierarchy); it did not cause them. The more particular circum- 
stances that Surrounded BSSR's presence in Portland are discussed In 
the following pages. 

The discussion of vouchers took place in the Department of 
Labor, and earlier in OEO and in Congress, w»*ere the Family Assistance 
Plan lived and died at the end of the 91st Congress. The initial 
Impulse for the voucher project came from the national office, which hired 
Leonard Goodwin, then at the Brookings Institution, to develop a concept 
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paper on voucher I ng. Goodwin was particularly Interested In the links 

among autonomy, self-esteem and employablUty. In administrative terms, 

vouchering potentially streamlines the delivery of services by condensing 

the roles of decision-maker and recipient (I.e., the client decides for 

himself what services he needs). In his 1972 paper on Introducing 

vouchering into manpower programs, Goodwin sums up the agency debate: 

A few officials and operating staff hypothesize that vouchers 
will bring no substantial change In the performance of enrol lee:. 
In present programs. Others see an Improvement In administrative 
functioning. They believe that vouchers will lead to the cutting 
of red tape In contracting services for enrol lees, and this will 
Improve the programs. 

A number of officials believe that giving trainees choices of 
training programs through vouchers Is good In Itself, furthering 
the general policy of decentralizing authority. There are associated 
hypothe$t;s that by giving poor people Increased control over decisions 
affecting their 1 Ives, they wl 1 1 have a stronger bel irf that their own 
efforts count. This belief Is then supposed to lead to better per- 
formance In training and a better chance of their obtaining and 
holding good jobs. The governing hypotheses for the experimental 
project are that vouchers will Improve enrol lees' belief In their 
own abilities, their skill level, and subsequently their work-force 
activities as reflected In Increased earn Ings . Subsidiary hypotheses 
center around the possible decrease In cost and Increase In efficiency 
of a voucher operation. 

Even while these hypotheses are put forward warnings are offered 
by still other officials that increasing the responsibilities of poor 
people who have little expertise In negotiating their own training 
will have negative results, even worse than at Present. The experi- 
mental project must test these kinds of prognostications J 

It Is Interesting to note that In Goodwin's discussion of the 
debate over vouchering, the discussion of practical administrative feasi- 
bility Includes hypotheses based on untested attitudes concerning poor 
people (e.g.. In paragraphs 2 and 3, quoted above), Goodwin seems to 
have assumed that cost arguments are not separate from behavioral argu- 
ments over what poor people will actually do with vouchers; however, 
they are separable, at least In theory. Administrators who approved 



Leonard Goodwin, 1972. Design of an experimental sti*dy for 
introducing vouchers Into manpower training programs. Final report 
submitted to Office of Research and Development, Manpower Admir Istration, 
Department of Labor, page k. 



of vouchering on cost grounds might, in fact be skeptical aLout the 
ability of clients to succeed with vouchers. Thus, support of the 
Idea of vouchering developed out of various lines of reasoning. The 
decision to go ahead with the project did not nean that all of the 
decision-makers shared uniform expectations of the research. 

In fact, skepticism concerning the ability of poor people to 
prepare themselves for employment runs deep at all echelons of WIN's 
administrative staff. Counselors and job developers were less uni- 
formly pessimistic than administrators at all levels about the possibilty 
of clients' succeeding with vouchers. Administrators appear to expect 
clients to exploit WIN, to sign up for maximum training programs, 
to take whatever they are offered. They also expect training vendors 
(schools and OJT enployers) to exploit the system. An irony is that 
these same people are In the business of providing the services 
they essentiallv blame clients for accepting, and buspect ot for _ 
offering. Thus, vouchering accentuates a philosophical dissonance 
that exists between these officials, views and their roles vis-a-vis 
cl ients. 

Again, the decision to proceed with the research did not mean 
that the people involved at th% Department of Labor had reached an 
agreement on what the priorities (cost versus clients' success) or the 
likely result of the research would be. Rather, it simply mant that 
they had resolved the lengthy debate over whether or not to proceed 
(albeit cautiously, with strictly-defined and limited feasibility goals); 
they saw their role. In fact, as decision-makers, preferring to leave 
the other questions to the researchers. This Is the earliest exanple of 
the unplanned dominance of administrative needs (I.e., ending time- 
consuming debate and limiting basic questions) over the substance of 
research needs. The question of which had the higher priority, p. /.serving 
tax dollars or increasing clients' autonomy, was a crucial one, yet, as 
It happened, it was not resolved at the outset nor at any time during the life 
of the research. The voucher project was to coexis* with, not replace, 
a program based on the traditional philosophy of giving higher priority 
to reduced administrative costs and quick placement and the debate between 
the two approaches was played out In a series of conflicts and adjustments. 
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At the Departmen'. of labor, the conflict was totally abstract, yet the 
present of even theoretical conflict had unfortunate effects on the 
research (as will be discussed In the text). 

There was a national search for a field site; however, Portland's 
eventual receptivity was similar to the Department of labor's In that 
It did not relate to the substance of the research. High-level local 
administrators met with the researchers to discuss the goals and design 
of the study, and were enthusiastic at the time. Further, one of the 
supervisors (whose contributions are discussed In the following sections) 
had expressed an Interest in voucher I ng to the regional office before 
the Bureau arrived with Its proposal. The decision to proceed In Portland 
was not contingent on the agreement or commitment of on-line staff; 
In fact, they were not aware of the early negotiations or the site search. 
They were fuliy briefed before the research was Installed In Portland, 
and again later on procedural matters related to voucher Ing. 

Thu-s, In the early days of the project, there were two crucial 
decisions that were made almost cot ncldental ly to the research as BSSR 
saw it and had proposed It: (I) the decision Zo go ahead with voucher lug 
was made without a felt need to make a concomitant convnitment (on the 
part of virtually all administrators) to what ft actually would represent 
in terms or administrative adjustments, and (2) the decision to accept 
the research in Portland was similarly superficial In terms of what 
was about to be attempted. This situation was 'lompllcated by the fact 
that the feasibility test contained some necessary ambiguity due to Its 
broadly stated goals, and that some of the staff felt they already knew 
what the outcomes would be. 

Fund Ing 

At the heart of the question of how the project was to be funded 
were (1) delay at the Congressional level and (2) a strong competitive 



The national offIce--and certainly the administrators ^elow the 
national level--had had very little experience with demonstration projects 
In tandem w^Ith on-going prog«*ams. In other voucher experiments, e.g.. 
Alum Rock, vouchering replaced the traditional method of assignment students 
to schools. The novelty of BSSR's research situation meant that no one 
Involved, from the top down, had had any experience that could serve as a 
guide in the events that ensued. 
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streak that ran between the various ?evels and components of the 
Department of labor. The effects of delay Appeared throughou't the 
life of the project, when It was periodically nece-sary to postpone 
program starts bee <use the shortage of regular WlH funds prevented the 
intake of new clients. An example of the workings of competition took 
place before Portland was cnosen for the study. An earlier search had 
produced S»n Olego, California, as a viable and willing site. Altl'iough 
the Department of labor originally offered to pay all costs resulting 
from the study (Including some administrative costs and tuition), San 
D'ego dropped out i4)en the California Department of Human Resources 
refused to contibute part of the state's new allocation to the project. 
The project monitor had Insisted on this point In order to save his own 
department's budget and as a matter of proper form, or ^'useful practice;' 
however, state officials had equivalent, strong loyalties, and refused. 

The subsequent agreement in Portland I was that Administrat ive 
costs and tuition be paid for by 001 and that social services (SAU) be 
underwritten by Salem The agreenent with SAU produced some later dif- 
ficulties 

Phast l_; Institutional Training 
Under Vouchers 

Although higher level agency staff had had advance exposure to 
the vouchtr project before it was installed !n Portland (but before 
research operations started), the local on-line staff did not. Its 
presence was annou^'wed In a memo--as one counselor said later, "like 
the thousands of others we get around here about new rules." The news 
struck different people differently, but It affected everyone, since 
the very concept of voucher ing touched on many. port ions of their 
routine duties. Under the idealized terms of the demonstration project, 
counselors, job developers, and SAU workers would all be put into new 
relationships with clients. 

The regular program responded to the new experiment that came 
Suddenly into its miost like antibodies react to a pathogen: after 
a certain amount of initial confusion and adjustment, the regular program 
began to change the voucher project in informal (but not subtle) ways 




into what was, to them, a mora usable form. In tha first phase of 
the research, this process of converting a parallel relationship Into 
a reciprocal one was hastened by the fact that the regular progr«n had 
no funds for Institutional training, while the voucher program had only 
Inst i«-utlonal faining funds. By the time voucher operations began, 
regular WIN funds for Institutional training had been depleted for 
scv;ral months. One aspect of the research design was destroyed by this 
situation since, instead of vouchers serving as an alternative to the 
regular orogram's resources, they now represented a route to unique 
resources: institutional training This was true not only for clients, 
but also for staff, who now had incentive to channel clients (or particu- 
lar clients) into the voucher program. 

Or ientat ion 

Before the first vouche*" was issued, BSSR arranged for training 
sessions by a team of consultants to orient the counselors to counseling 
m a vouchering setting. 8SSR trained the counselors on procedures. The 
researchers expected that a major change for the counselors would be in 
what they and the consultants called "nond irect ive counsel ing.'* ^ The 
basis for this style of counseling was self-assessment by clients, a 
process by which they recorded their own i'^terests, goals and talents 
as they saw them, presumbly as an aid in mustering their own resources 
to find training or employment. Some of the counselors felt that they 
already had the skills the professional training team was preaching, 
and that they already used a nondirective approach with their clients. 
Furthermore, the counselors already knew each other and their counseling 
styles, they knew no amount of training could make all ten teams work 
in exactly the same way. Skepticism and self-knowledge inclined some 
of the staff to reject what they heard. 

The job developers and SAU workers--who were not trained by the 
consul tants--said that they should have had some orientation, since their 
relationships with clients would also be somewhat changed by the presence 



^ Later, BSSR was to regret having used the term '*nond i rect i ve" 
at all, since it created some resentment among the counselors, with whose 
traditional style of relating to clients it was meant to contrast. 
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of th€ voucher option. They felt that they w^re left in the positi 
of risking being at cross-purposes wUh rhe WIN staff. For example. SAU 
workers wondered to what extent ihe^ should i^ithdraw from decision- 
making that was peripherally related to training, on behalf of voucher 
c1 ients. 

The training sessions were the first opportunity that staff had 
to see that the researchers' (and, by implication, the Department of 
Labor's) perceptions of their rol«s were different from reality as they 
saw it. The training (which did not emphasize nond i rect Iveness , but 
rather self -assessment) , was aimed at • group that did not want training 
»t all, although BSSR assumed they needed it. and ignored the groups who 
felt the need for orientation, but who had none. Although the sessions 
were daslgned to equalize the differences among the ten teams. In fact, 
they served an entirely different purpose: a tap on the wedge that 
already existed between WIK (and the project) and SAU. 

Voucher ing 

The advantage to the counseling staff in cooperating with the 
project was that the voucher project was amply funded-unfortunately 
in contrast to their own program. (Although everyone involved knew 
that the voucher project and the regular program did not compete for 
funds, it was difficult to prevent the emotional reaction to what 
seemed like an unjust division of "the tax-payers' money,") Thus, 
counselors were grateful for the resources that the project offered 
them, although they did not fully accept its terms. 

To ^ .se counselors who felt comfortable with the idea of voucher ing. 
the project seemed like another version of their regular program roles. 
'Nt makes formal what we do informally all the time." one of the local 
staff said. Of course, in the minds of the r^earcher^. it was not 
supposed to have been a replication of their ordinary routine. In 
retrospect, one interesting question becomes how it was possible that 
researchers, high-level agency administrators, and on-line staff could 
have such different perceptions of staff roles. Researchers now say 
that they shared a stereotype of a WIN counselor as a paternalistic, 
•uthorltarlan-al though wel l-mealng-person with Goodwin and Department of 
Labor officials, that their view of the real workings of the regular 
program was Inaccurate because of this. Like many others, they assumed 



a fairly cfosa ralaticnthlp batween pfocadurai and oparatfons. Tha 
ragular staff felt that the rasaarchars expected them to be "oppressive** 
or "punitive" (In staff's words). Staff rebelled under that view; to them, 
the irony was that the BSSR's proc^-^Mres were more restrictive than 
(unwritten) ones they normally followed. The most unacceptable aspect 
of the voucher project to them was that, for research purposes, some clients 
would have to be al Towed to fall. What the reseerch celled "self- 
determination," the staff felt was closer to ebendonment and negligence. 
While staff felt that the "Eastern liberals" (their term, again; In fact, 
BSSR's project director wes raised ne^^r Portland) on the reseerch staff 
had designed the project as they did beceuse the counselors were not 
doing their Jobs well enough, they felt that the Bureau's experimental 
program had the potential to penalize clients. 

What loolted to BSSR as the attempt to provide a test of one form " 
of a voucherfng system, looked to staff as advocacy of the system, in 
fact, the closer the day-to-day functioning of the voucher project came 
to fulfilling the goals of the counselors, the more difficult it was for 
the Bureau to assert itself on its own goals. 

Clients' needs versus reseerch needs . -^-Thg first voucher was 
issued on April 29, 197^. By the end of the first week of vouchering 
a number of issues were raised that were to remain with the project. 
Briefly stated, these were; 

— cov-selors* fears that Individual clients were not "ready" for vouchers; 

—counselors* stretching search time by issuing a voucher after 
the search had already begun; 

—counselors' offering (or fK)t offering) vouchers, as they saw fit. 

In addition to these procedural problems, other problems arose: 

—fear of a law suit by clients who did not understand the voucher 
process end felt ebandoned, or by clients who d»d not receive 
vouchers; 

--SAU's feelings of alienation. 
Presumably, the deviations from the procedures all were made In the 
interest of the clients; however, they went to the heart of the experf- 
menta) design of the research. This conf I let-In the eyes of counselors, 
a conflict of research versus clients-was to become a theme as the 
research progressed. Clients* choices, after all, were much less reveal- 
ing once a client had been screened by his or her counselor. A test 
of the feasibility of a 5 ix-^eek search time could not take place If 
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counselors stretched it for clients* comfort. Success with vouchers 
could not be measured accurately If the clients' experience was altered 
by uncooperative or confusing support services delivery. 

The structural needs of the research, therefore, came into 
conflict with what counselors saw as the clients* needs. Staff regularly 
adapted the regular program rules on behalf of the clients, and iW- 
diately began to do the same with the vouchering procedures. The results 
were that (l) the demonstration could not test what it was designed to 
test, and (2) in some cases, the regular and vouchered clients' experi- 
ences with the program were indistinguishable. 

Agency needs versus re search needs . ^-Al rhnur^h client benefit was 
a major concern of the counselors, they also had other, organizational 
imperatives which counter-balanced the project design. This was because 
the procedures altered (at least in theory) counselors* relationship to • 
clients without altering their (counselors') relationship to the agency 
(WIN). BSSR's assumption was that counselors would not be responsible 
for clients' employabi I ity and placement under vouchering; however, 
counselors were accountable in the agency's view. This contradiction 
may have reflected substantive misunderstanding of vouchering on the 
part of state agency administrators (who. of course, had their parallel 
accounubility to the regional and federal offices to be concerned 
with). The contradiction produced a series of procedural modifications 
beyond the one mentioned earlier: 

-counselors filled out Basic Training Agreements with clients 
to shepherd them through the process: 

—counselors trier* o avoid having to modify or Justify post 
facto the Employ i 1 1 ty Development Plan filed with SaT^ 
(the state office), by encouraging clients to stay with one 
plan. 

In terms of the research, these changes meant that It was increasingly 
difficult to determine when a client was acting on his own behalf and 
when a counselor was intervening for him. 

For example, one state administrator said to me, "Except for 
ncndlrect ional (sic) counseling, vouchering is great." However, what he 
failed to see was that counseling was an Important aspect of vouchering- 
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The Structure of Staff Response 

Counselor's response to vouchering cen be seen as a bureaucratic 
one- -they reacted to the new procedures In ways that had been structured 
by their prior experience with WIN. Some of the more profound Instances 
of this pattern have just been discussed, I.e., the discomfort counselors 
felt with clients making decisions on their own behalf (even though 
counselors say that their clients very often make their own decisions 
under the regular program as well); «nd bending the vouchering proce- 
dures on behalf of the clients, just as they bent the regular procedures 
The regular WIN program also structured other responses, however, for 
example, while BSSR saw sel f -assessment as a process, many of the counselors 
saw it as another form (it is in the form of a questionnaire), simply to 
be completed. One counselor felt that, to clien*:s, going through self 
dSies^iment was "not worth the time, once they have their money" (i.e., 
the'r voucher) This was a common, mistaken view of the relationship 
of self-assessment co vouchering. 

Early in the life of the oroject, then* one could begin to see 
two modes of dealing with vouchers begin to develop. The Bureau's 
mode was in terms of research needs: while maintaining tNi discreteness 
ef the regular program* add another alongside It aitd measure the dif- 
ferences in their operation. One reason that vouchering proved so 
illusory in practice was that the regular program could not maintain 
its ov^n boundaries with the new project adjacent to It. 5 Because the 
regular program staff related to i^Ibuchering In their own mode, which 
was in terms of their own time allocations and their need to husband 
scarce resources, the voucher project could not be left on the other 
side of the membranous agency wall; rmthen It was absorbed and some- 
what uncomfortably digested. 

The breakdown of part of the research design, and goals tended 
to undermine the confidence of the agency staff in the project and its 
claims to scientific integrity. In other words, they did not feel 
bound by the "rules" when, from their point of view, they did not work» 
or tended to work against clients. This sort of f rus trat ion was not 



^ One might well argue--as some counselors do In retrospect-- 
that the boundaries between the regular and voucher programs were 
never very clear In practice. 
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directed at BSSR except insofar as i t was the vehicle that Imported 
the problem. Especially for counselors who felt that nondirective 
counseling was the same as abandonment, progran^tic restrictions on 
wh«.t they could offer their clients became a real thorn. These restric 
t.ons-e.g., no training longer than 12 months-were no greater than in 
the.r own program; however, the expectations were higher since vouChe^ing 
was new and coming from a different source. In fact, however, it often 
seemed that vouchering did not get at clients' real needs anv more than 
WIN II did. Thus, the reaction of s«aff to BSSR-s project wa, structured 
not only by what exuted in the regula, program, but by what did not 
exist, as wel I. 

trilateraT 7"" °^ 

at r , and somet.mes confusing dialogue anx^g the counseling staff. 

BSSR ,n Portland, and in Washington. The dialogue was not wholly a 

from BSSR.s po.nt of view, the sequence of me^s appeared disjointed, 
and sa™.t.mes superfluous, to the staff. Often, memos described pro- 
cedura correctives that had already been tried and discussed. They 
Hrlad T ^ °' wer 

and ad p .vely to a new situation, they did not always perceive the 

Portland and Washington was doubly confusing. 

In general, communication between the counseling staff and the 
-eau.s staff was informal almost to the point of being accidental 
In casual conversation, field staff discovered facts with tremendous 
"npl.cat.ons for the research. For example: 

"cZ'Me"err"aSrf^'"'K°'J::"' '° ^ counselors 

example of ^ eL .r^^li 

vouchers- tte olH V I structuring the response to 
to allow job-ready clients to accept vouchers); 

"t;\":^c;::°:r:as"^^sTic;:drf:*'Tr ^ "'^"""^^ 
their counse,o^^a:"n:r:o^":L' c'^ n^^r'."::'" 

reregistered in the progr« with a new record! 

"J^h^Hnri^Ln^i^cludlnr::^ °' ^►'^ 

assessment; '"c'^ding the voucher itself, and self- 
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— counselors were saving thelV vouchers for special clients;^ 

--•t least one counselor would not give a voucher to someone 
whose prison record she felt BSSK staff should not sea* 

Contl.'tuat ion WLthout Resolution 

As the study continued, its boundaries, such as the one just 
described, became less certain. Given the range and depth of counselors* 
attUudes toward vouchering In practice , not to speak of their ordinary 
routines, U Is not surprising that the ten voucher teams performed very 
differently. As we have seen, the teams Informed their clients of the 
availability of vouchers In different ways with varying amounts of 
encouragement. The result was that ten more or less fully screened 
(on different bases) groups of clients found their ways to vouchers. 
Instead of one voucher study, there were "ten tiny studies," ^s one 
researcher said later, with cLient groups so distinct that they show 
up as statistical entities in the empirical analyses of the program 
(see, for example, 'HJccupat lonal Choices and Vocational School Selections 
by Bruce Dunning, 1976). 

Briefly stated, the voucher program and the regular program did 
not find a particularly easy mesh, although the staff came to be fairly 
enthusiastic— especially about the abstract value of vouchers. The 
strains between operational and research objectives were usually 
tolerable. As one counselor said, "Institutional vouchering was 
successful and Interesting, but not perfect." At the beginning ot the 
research, 8SSR*s presence was presumed to be temporary. According to 
the original design of the project, the feasibility experiment was to 
conclude when 150 vouchers had been committed for training. Originally 
there was no Intention of foliating clients through their training or 
Into their first efforts to find Jobs. The research later shifted In 
emphasis and became more broadly focused. As It happened, the Bureau 
6 ~ 

Initially, 150 vouchers were distributed to the ten counselors 
equatty, each had a ration which, when depleted, could not be restored. 
As mentioned above, some counselors responded by saving the vouchers 
for people they felt would bo especially successful with them. This 
practice was directly contrary to the research goals, since the experi- 
ment was designed to test demand on the part of clients, not counselors. 
When BSSR realized that rationing the vouchers was havinq this adverse 
effect, It modified the procedure and pooled the vouchers (de-rationed 
them), for the counselors to draw on at will until the whole supply 
was committed. This modification recced the incentive to hoard the 
vouchers and control their flow. 
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was not gone from Portland at the end of the I50 vouchers, but stayed 
to study on-the-job training. The politeness and cooperation that were 
extended to the research operation when it appeared only to be a short- 
term event deteriorated as its presence continued, apparently indefinitely. 
This deterioration was also a result of other factors, as we shall see 
in the next sections. 

Phase 2; On-the-Job Training 

In August, 1974, at the Portland I debriefing. Department of 
Labor officials initiated discussion of a second experiment that would 
offer institutional training and OJT. The extension of the voucher 
study to on-the-job training (OJT) had not been contemplated in^the 
original grant from DOL, but instead developed in response to con- 
clusions that vouchering institutional training had in fact largely 
been administratively feasible. The local superv isor was extremely 
enthusiastic; the Salem officials, less so. but cooperative. During 
the next seven months, wiiile the agreement was negotiated, available 
funds molded the research design in important ways 

First, the situation with SAU had to be resolved, $AU was asked 
to give estimates of its anticioated costs. In doing so, its supervisor 
incorporated-on€ iiundred percent client need for child care, with costs 
estimated at $1000 per client for all support services. These estimates 
were not challenged at the time; however, the actual rate of client naed 
for child care was closer to twenty percent of the total clients. The 
budget sent to the Department of Labor Included the SAU estimate, plus the 
estimates from Portland, Salem and Seattle for 300 open vouchers (i.e., a 
choice of institutional or on-the-job training), with a two year (instead 
of the one year limit from Portland I) limit to Ihe training. The total 
came to over $1.3 million dollars, which exceeded available DOL funds. 
Research costs were cut back, the training time was put back to its 
original one year limit, and institutional training was removed from 
the program. These reductions brought the budget to an acceptable 
level; with research costs, the Federal outla-was approximately 
$900,000 dollars. During Portland II, SAU estimated its need at $200,000, 
overheaded at ten percent of the dollars spent, i.e.. SAU could collect 
the overhead only on monies it had already spent, not in anticipation of 
expenditures. This meant ihat although the program had ample funding 
to run on, it had no additional staff to administer the funds. (ES, 
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on th« othT hand, had laparatt butfgat Ifnti for Its subsidy and Its 
ovtrhaad, and could add n«w staff.) Tha Imbalance batwaan snpla funding 
and short staff bacima chronic In SAU, Tha situation lad to chargas that 
tha voucher project was causing an excess of work among SAU workers 
(although this was not a unanimous charge). 

In the meantime, however, SAU funding had bean drastically 
cut back by Congress, and regular SAU In Portland shut down completely 
In November, ]37k. Nobody who needed child care or medical care could 
enter WIN during this budgetary famine; only direct-placement clients 
without babysitting needs and who were healthy entered the program. 
This situation produced the possibility that vouchering would open, 
once again, offering a unique resource, child care. Indeed, vouchered 
OJT was the oalx way Into WIN for mothers with chl Id-care needs, unless 
they were placeable; thcr would be no volunteers under these conditions. 
The Ideal research situation would have bean ons in which the vouchers 
led to the Identical services and funding levels as the regular program, 
so that the appeal of the vouchers themselves would be measured without 
being complicated by the allure of these other elements. 

In addition, regular OJt funds dried up before vouchering 
could begin. One sympathetic Salem official offered a small reallocation 
to Portland to keep the 04T option j live Jo r the sak e of the research; 
however, the money arrived and was spent before vouchering could 5tart up. 

In April, 1975, SAU was partially reopened to provide for short- 
run child care for direct placements. The research contract came through 
at the same time, contributing $203,000 to a special SAU fund for services 
to voucher cl ients . 

When vouchers became operational in early June, 1975 , some of 
the problems had been solved, but others were quick to develop. SAU 
had been restored to part of its normal funding strength. ReguL^r OJT 
funds were expected in an Innlnent Congressional appropriation. SAU 
had befn awarded a special fund to facilitate the provision of services 
to voichered clients. The research design had been modified to acconwo- 
date|to budget constraints. The new problems were that within a month 
of jj^e start-up of Portland 1 1 , OJT funds had not yet arrived, and regular 
SAU once again shut down. The voucher project was once again In the 
position of offering a unique (not simply a parallel) option: just as 
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clients had had to accept vouchers td obtain institutional training 
during the first phase, clients who needed social services during the 
early parts of the OJT phase had to come through the voucher proces 
to have^access to it. 

Through informal communication networl^s, the B$$R field staff learned 
of one source of the delay in regular OJT funding: the money has been 
pooled to the rest of the state, leaving vouchers the only route by 
which clients could have access to OJT in Portland. This decision hac 
been made in Salem in order to protect Portland from the consequences 
of a double allocation (i.e.. money competing for consumers); an excess 
of regular funds at the end of the year would have repercussions during 
the next fiscal year. Also, administrators presumed that Portland's 
labor market could not absorb a doubled number of OJT contracts. The 
research implications had not been considered; however, the same man 
who obtained ten thousand dollars for an OJT tranfusion earlier in 
the spring now promised to restore regular operations funds to Portland 
to insure a full comparison. He eventually did so, but by the time 
the reallocation arrived, weeks had gone by without a regular program 
to balance against the use of vouchers. 

Regular SAU shut down for the second time on June 30, again 
reflecting budgetary unc^rtaTnty that prevented it from repaying the 
previous year's deficit. This problem as well as the shorta9e of OJT 
funds had stemmed from delays in Congress over HfW-DOl appropriations 
in FY 1976. In a policy of caution, the region kept a wary eye on any 
expenditures made in anticipation of th« new year's budget. This situa- 
tion continued over the entire life of Portland M: both the researchers 
and the agency's administrators were paralyzed by it. 

There were some other important aspe'-*s In the OJT phase's 
background. First. OJT involved job developers to a much greater 
extent than institutional training had. Committing an OJT voucher 
Involved being hired for a job. and was treated in program reporting 
statistics as a placement. Second, for the Job developers especially, 
OJT involved heavi ly-laden responsibilities. Their regular roles 
involved personal relationships with groups of employers with whom 
they had a two-way unstated arrangement: the job developer would provide 
a well-qualified and reliable client, and the employer would agree to 
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slgn his contract. To fulfill their side of the bargain, job developers 
felt justified in screening their clients for the most likely candidates 
for successful hires before tponforlng them. Under the vouchering 
procedures, staff were expected to facilitate the completfon of the 
contract (but not negotiate it) without prior screening. From the 
researchers' point of view, this was the equivalent of the staff 
role during the earlier phase. To the job developers, however, it 
meant standing behind an unknown client and risking the working 
relationships that they had worked hard to develop with employers. 
Thus, from the beginning of the OJT phase, there was strong inducement 
for Job developers to intervene in the cqnmltment process. 

Third, as there had been during institutional training under 
vouchers, there were placement pressures coming from WIN that worked 
against the Idealized freedom that the vouchers promised. In the 
case of OJT, pressures were much more severe, since under the Social 
Security Act. one-third of all WIN ex^ndltures must be for OJT or PSE. 
Agency pressure constituted a second major force thAt placed job developers' 
professional self-preservation at some disparity with the research goals7 
This is an example of how SSSR's temporary and tandem position In Portland 
led to less cooperation than there might have been under other circum- 
stances, e.g,, as a permanent operation. Job developers were not likely 
to jeopardize their own career development and that of their clients for 
the sake of the visiting researchers. 

In addition, apart from the structural difficulties of synchro* 
nizing the two programs. OJT was especially badly timed because of 
the economy. Unemployment was at record highs across the nation and 
placements --and not only for WIN cl lents—w^re extremely difficult. 
Later, some administrators blamed the voucher project for the decline 
in placements during this period; however, staff did not appear to 
make that charge. 



7 

The National WIN office offered to arrange a temporary status 
for Portland that would have relieved the program of placement pressures 
by carrying over the previous years placement figures for one more 
fiscal year. This would have assured WIN of meeting its placement 
goals and simultaneously would have removed Incentives for job developers 
to unilaterally modify vouchering procedures for this administrative 
purpose. The local office supervisor, however, refused this offer, 
probably out of pride, but also out of the conviction that vouchering 
could 'work." Thus, staff and researchers were left in a situation 
in which each saw their own goals threatened by the other's procedural 
or administrative needs. 
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Once vouchering was under way in early July, 1975, a pattern 
of adjustment, similar to that described for Phase I, began to unfurl, 
complicated by the 'tensions just discussed. Of course, with no regular 
P'-Cy.am child care, accurate assessment of client demand for vouchers 
(as opposed to the services attached to them) was made extremely 
difficult. Another early event was the accidental discovery that 
Portland^ regular OJT funds had been reallocated to the rest of the 
state at about the same time vouchering was starting. An administrator 
had promised BSSR that during this phase of the research, he would 
guarantee that a full comparison of the programs would be possible, 
and he fulfilled that promise by reassigning funds to Portland, thus 
solving the problem. This story is interesting for two reasons; (1) 
it illustrates the tension between higher-level administrative needs 
(i.e., budget) and research needs; and (2) when counselors refer to 
this episode in retrospect, they do not add that the problem was 
resolved. Instead, it remains for them yet another of the blows that 
they consider the voucher project to have dealt them and their clients. 

Start-Up 

The SAU shut-down and the confusion over regular program funds 
both occurred around the time that vouchering began for Portland 11. 
In this sensitive climate, job developers began their work, somewhat 
caught between the voucher project and their agency, Salem was known 
to be sensitive to employer exploitation of voucher clients; an exces- 
sively long proposed training time on an OJT contract could cause a 
delay of weeks before the client would be cleared to start the OJT, 
(Clearance required communications between Portland and Salem, Salem 
and Seattle, and back.) Field staff began to report ;50me undercoding 
of occupations in order to expedite the processing of clients' papers 
by reducing the length of training? 

o 

Coding refers to the Occupational Rating System (ORS) , which 
lists occupations and the maximum length of training permitted. Vouchered 
OJT was originally designed with a blanket limit of 52 weeks of training; 
however, when the first f^^w contracts were all written for 52 weeks 
(including two for produce stackers), a "maximum" permissible time for- 
mula was devised, based on training times derived from the Dictionary of 
Occupatio! 1 Titles (DOT). The actual formula was DOT + 8 weeks. 
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Most of the events that disrupted Portland il were not of the 
sort that occured throughout Portland I, internal structural adjustments 
to the new entity (voucher ing) . Rather. OJT was affected more by 
external political and administrative events. These, too, had an effect 
on the local staffs and their clients; further, staff began to perceive 
that BSSR was not an autonomous agent in Portland, but was — like the 
staff--5ubject to being buffeted by outside forces. One of the earliest 
of these events once vouchering had started was the decision to enforce 
an existing ruling that prevented applicant AFOC fathers with pending 
Ul claims from registering with WIN. In terms of research, this provi- 
sion introduced an element of. noncomparab 1 1 1 ty with earlier male, non- 
vouchered OJT c!lents. The rule also appeared to exclude from the 
potential vouchered population the men who were among those most 
likely to succeed at OJT. i.e.. those men with fairly recent work 
experience and with the desire to secure additional training. 

Other events had to do primarily with money and staffing, and 
the awkward balance between them. COft^ress lonal appropriations for 
Fiscal Year I976 were badly delayed, and uncertainty over the appropria- 
tions for WIN made administrators very cautious about releasing funds 
for any purpose. SAU was not in a position to recruit new staff, but 
also could not pay for child care unless It was at a certified center 
or home (following a national requirement). Under these conditions 
local SAU would not provide child care for vouchered clients, despite 
the availability of separate funds for these clients. As an accoiwnoda- 
tion. the local WIN supervisor offered to the local SAU supervisor to 
control enrollment of clients with child care needs in the regular WIN 
program; much later, setting limits was discovered to be illegal. For 
months. SAU and the local WIN supervisor juggled staffing possibilities 
so that the voucher operation would have enough manpower; however, this 
situation was never resolved, except temporarily. 

In the crunch caused by the downward spiral of declining 
placements, precarious funding, a poor economy and agency pressure 
for placements, the local staff became increasingly sensitive to the 
presence of vouchering m their midst. In retrospect, they now say 
(and apparently said then) that the local prime mover behind vouchering 
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v«$ responsive for giving it priority over the regular program. 
Although He was weli-loved by his staff, more than one person referred 
to his having "sacrificed" the regular program for the sake of the 
study. It Is difficult now to reconstruct what form the so-called 
sacrifice took: most staff seem to feel that he had withheld the 
psychological support that they felt was necessary for their own 
successful job performance, that his attention had been diverted. 
Most of the regular teams place a great deal of importance on the 
pressures to which they are subject. In a positive sense: they speak 
of inter-team conpetition and the promise of rewards for high placements 
as being real incentives for successful work. These same people seemed 
inclined to feel abandoned when their supervisor showed interest in 
what appeared to be a "riva/* progr«n. 

A Finai Cr Is Is 

In the late fall of 1975. toward the final weeks of OJT. two 
dcvclopnents occurred that brought voucher project relations with the 
regular program to a head. First. th« ten-team approach to voucher Ing 
finally appeared to be at crojs-purpoiet with the research. Of course, 
this idea had been discussed since the beginning of Institutional 
voucherlng. when the ten teams handled the progrm option so differently 
from each other. Under the Intensified pressures described for OJT. 
there was severe deterioration of any semblance of uniformity, and 
norcooperatlon clearly affected the research. Ore specialized te«n 
replaced the voucherlng functions of the ten Janus-like teams (one 
face toward traditional service delivery, one face toward vouchered 
delivery). This move was demanded by some of the teams, but for 
others . the creation of one team only emphasised what was to them the 
elite nature of the voucher project. The new voucher te«n had I ti own 
funds, its own room, and specialized roles. One supervisor looks back 
on the change as healthy one that insured his office's cooperation; 
another supet . Isor says that the change damaged the democratic structure 
of local WIN and that his staff's morale suffered as a result. One staff 
member said that the creation of the speC.^I team meant that voucherlng 
had become an effective escape from overwork, since -llents who chose 
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• voucher mr% removed from the reguUr fm%^ caseloads. In any c«s«, 
the nove from ten teams to one resolved some of the research dlimMS 
but certainly created others, due to the fact that staff did not react 
uniformly to either the problem or Its solution. 

In the case of the move to one voucher team, administrative 
and research goals again came Into conflict. The research goals were 
eiear—ten teams with varying degrees of cooparatlon and comprehension 
were not a satisfactory base for essessing the feasibility of voMChering; 
however, the staff was not told jgfhx the consolidation was taking piece. 
From the administrative point of vlew« the move to a new team translated 
Into the ablUty to hire more staff, as long as the special fund remained. 
I^al WIN and SAU supervisors were asked about their staffing needs; SAU 
replied that five to nine new people would be needed to help with the 
additional work caused by the voucher project, in the end, people 
were found, but red tape and unpredictable events pregnancy and 
resignation from one and an Injury and disability In the other) pre- 
vented the transition from being an easy or quick one. 

In the meantime, the other development that tore at the rela- 
tionship between the resear'ih and staff was the sudden Illness and 
death l^ late November, 1975, of the supervisor who had stood behind the 
voucher project. Some angry itaff blamed his hnert attack on the 
stresses caused by the research. The coheslvenesi In the operation 
of both the project and the regular program that had leen an effect 
of his personality and popularity fell Into disarray as soon as he 
died. The supervisory $taff was reorganized following his death, and 
hli replacement's Inclination (and wish) was to turn his back on 
vouchering and restore order to his agonized itaff. ThU angry mood 
filtered upwards through the levels of the agency; Ironically* the 
staff new seems less angry than the administrators who claim to speak 
for them. 

Also Ironically, the voucher project was caught with a surplus 
of funds toward the »nd of Portland 11. Researchers had realized this 
possibility during the summer, when they discovered that SAU's estimates 
of Its needs had been toe high. Furthermore, there was an excess of 
unobligated funds due to a low convnitment rate from ES as well. 




-22- 



Various plans canM under discussion for the disposition of thesa funds. 
Th«ra ms soaw discussion of starting nmt projects In Eug^a and/or 
SalM, but thasa nmra rajactad aithar by ISSR or at tha stata laval. 
Tha question of Out to do with tha surplus was not rasolvad until 
aftor tha WIN supervisor's death. 

Phese U Open Voucher Ino 

Slmuitaneousiy with what appeared to be a hrutai and to vouchering 
In Portland, negotlatlona over one more phese were InltiaUd by the state 
•dninlstrator who had been so supportive during Portland li^s financial 
diff Icuitlai. He proposed what was to become open vouchering, I.e.. clients 
choosing not only between the regular program and vouchering, but between 
institutional training and OJT. Phese 3 would be the first full com- 
parison.^ 

Tha winter of 1975-1976 was spent developing the local, state 
and national admin Utratlve arrangements for the new study. The 
initial difficulty was In securing the cooperation of SAU. The rela- 
tlonshlr betVMen WIN and SAU had deteriorated badly i t every administra- 
tive level,^d SAU was at first unwilling to proceed with another 
phase of the project. The Regional office said that it would not forward 
Its proposal to Washington unless SAU's anxieties were relieved. 

Once again, calls went out to solicit estimates of needs for 
what was to become the short-lived Ptorrland III, This time, state 
SAU registered Its protest by slowing down its response. The region 
finelly forwarded the proposal to Washington early in January, I976. 
The next six months were a series of delays; every communication had 
to filter up and down the system before progress could be made. 

9 

Today, It is uncertain whether staff knew (or remember) that 
the three phases were not planned from the beginning of the research, 
but were rather unanticipated outgrowths, Thus, staff sees ISSH's 
relations with a^tnlstrators In a somewhat mora strategic light than 
!s actually correct. 

10 

\^ucher funds t#ere transmitted via two contracts. Each 
^•>*n9« (1-^ llj 11-^ III) required time-consuming negotiations 
among these various levels of the WIN progrM. 
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When the contract c«ne through, th« •llotment for SAU w«s high 
relative to past expenditures; however, the supervisor w«s stMl uneble 
to hire steff to edmJn Ister the office's funds. The quote that emerged 
for the first time in the letter of agreement (discussed below) prevented 
the surplus from being used for vouchers. 

In late May, Portland Hi was ready to start, but once again, 
it was severely altered by external events. This time, they came 
unexpectedly from the Department of labor—previously considered an 
ally of the research b, the local and regional staffs. Contrary to 
wha*- had been agreed upon during the negotiations over the grant 
modification, the letter of agreement between the Office of Research 
and Development (ORD) and Region X in Seattle specified a 50%-50% 
division between institutional train Ing and OJT. ^ ^ (This percentage 
parity was later translated into a raw number quota of kS/kS, since 
Salem estimated that ninety contracts could be drawn from the remaining 
research funds.) The establishment of this quota was Important In 
understanding the last weeks of the research for two reasons. First, 
it was a surprise to BSSR, and oubllcly known to be. The Imagined 
alliance of the researchers with the national office had been an 
important factor in structuring peopte*s reactions to the project. 
New that that alliance had been made ambiguous, the research took on 
a new face, apparently--judglng from the rate of cof»peratlon with 
Portland Ml --a less important face. Second, the quota affected the 
research, in that it reinforced the tendency that developed among staff 
during Portland I (under the ration system) and Po'-tland M (with high 
placement pressures) to screen heavily. Under the quota system. It was 
doubtful that client demand for OJT versus Institutional training could 
be ascertained at all, and» even If it could, It was even more doubtful 
that the demand for vouchered Institutional training could be satisfied. 

A secondary result, although one that perhaps had the greatest 
impacc on the resear-h. W4 that the field staff was put In the position 

M 

The reason for the quota was that a quota nad been agreed upon 
for similar research that was beginning in Baltimore; the national office 
simply followed the same mode! for both sites, although there was no 
necessity for the research conditions to be the same In both places. 
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of having mad* false promisas to tha counsalort and Job davalopars. 
Ona incfdant mada this a tarious situation: ona offica had baan 
stockpiling cliants In anticipation of mora vouchars and, ««han ot ar 
off leas protastad, fiald suff thought thay wara honestly raassur.ng 
tham that there v^uld be plenty of vouchers to 90 around. Mot only 
were there, in fact, very few vouchers, but since the more popular 
Institutional vouchers were likely to be used up quickly, order of 
arrival at the voucher office was also a crucial factor in clients* 
"stand-by** status. 

The staff was outraged, not at the Sureau—since now it was 
apparent what BSSH, too, was a victim of sorts— but at their own 
bureaucracy. Their response. In spite of field staff's efforts to 
cajole, was to boycott the voucher project. They felt that the vouchers 
were too unrenab!e a commodity for them to recommend to their clients 
since already they had sent clients interested in school training to 
an orientation, only to find that institutional vouchers »^re no longer 
available. They were angry on their own behalf, and on that of clients. 
Ironically, Portland Ml existed at all partly because staff semi- 
cooperation had had the effect of slowing down the voucher ing process 
in Portland II; new noncooperat Ion was virtually complete, with many 
counselors not informing clients of the voucher option, and voucher 
registration down to a trickle. 

The Final Situation 

Given the field situation, it is difficult to assess the effect 
of the voucher team itself on the research. Although they t^ere housed 
in the same building as regular WIN staff who have their own supervisor, 
the voucher team was not under the direct authority of that supervisor, 
but of the next higher-ranking person, whose office was several miles 
away. Thay ware thus segregated administratively and physically within 
tha building, since they had their own large room in the basempnt, 
while the other staff shared rather small offices on the first floor, 
Tha BSSK staff was also In the basement, although at the other end of 
the building. The effect was a considerable degree of physical and 
managerial autonomy for the voucher team, and, in the minds of the 
regular WIN staff, there was a natural (but inaccurate) association 
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of the voucher team wUh BSSR, Generously, It can be satd that the 
team exploited its Independence to a degree that should be considered 
a factor in the experience of clients with the last phase of the project. 
They were absent from their desks for large portions of the day. leaving 
clients neglected for long periods. When they were present, the atmosphere 
in their office was one of clowning and disrespect. Playing games and 
lounging at their desks in the voucher team office were not interrupted 
even when the supervisor was present. It is possible to interpret" the 
lack of management of the voucher team as a display of cy^niclsm — or even 
hostility— on the part of their supervisor. He stated tha\t he expected the 
voucher project to "fail" (his word), and apparently made lUtle effort even 
to maintain its integrity, 

Portland 1 ir ended far short of its goals; in fact, without ever 
having truly begun to operate. Given the quota, the opportunity for 
open vouchering never really existed because of the stockpiling described 
above; even If that had not occurred, a real choice among training routes 
would have survived only about one wsek before the institutional quota 
began to eliminate an option. Oregon has been able to keep the money 
that would have been used for vouchers, and allocate it as it wishes. 



The Bureau *s staff did not oversee or direct the voucher team; 
the latter were WIN employees, 

^ For example, on one day when I was present, the clerk/recep- 
tionist wore Mickey House ears while greeting clients. The office walls 
were decorated with posters and cartoons of, at best, locker room quality. 
Although as individuals face-to-face with clients, members of the team 
acted with some dignity and seriousness, they did seem to have lost their 
sense of what the carnival atmosphere In the room did to the credibility 
of their effort. 
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CONCLUSION 



This report is hindsight. It should not be read with the preface, 
••If only we had known, . because It is not possible to say with any 
certainty which specific events would have been different (and In what 
way) under different circumstances, administrative and otherwise. On the 
other hand, It Is equally clear that some events were produced by the 
structure of the situation, whereas others were products of Individual 
personalities and Idlosyncracles as well as unpredictable external events. 
In this chapter, we turn to the former, that Is, to the kinds of events 
produced by the very organization of the relationships created by the 
research from Its genesis to its reporting. Treating these events as 
variables raises some questions that go well beyond the history of this 
one project; they are questions that go to the heart of evaluation research. 
In this chapter, we will begin with some structural generalizations from 
the Portland experience, and proceed to broader research Issues. 

Staff Support 

Throughout Portland 1 and It, a major variable was the level and 
nature of staff cooperation. As the chronology of events shows, the 
voucher project was accepted in Portland without the prior knowledge or 
consent of the on-line staff. Their own supervisor and the administrative 
hierarchy above him were fully briefed, and the prospect of additional 
dollars and job slots elicited some enthusiasm. In retrospect, some 
staff members now point to their lack of participation in the final stage 
of the site search as an indication of the disregard with which they claim 
to have been treated by the researchers. On the other hand, the same 
Individuals remember that when they eventually did recelve'iniiiios detail- 
ing the procedural adjMttmants the research would Involve, their reaction 
was matter-of-fact, since the voucher project memos were like the "thousands 
of other memos we get around here all the time." The consent Issue was, 
perhaps, a convenient peg for expressing the deeper frustrations that 
thi report discusses. Thus, even In retrospect, It Is difficult to 
assert that prior consent of the staff wOuld have eased any of the 
angry feelings that developed later. 
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In general, though, the episode raises the larger question of 
whether or not staff should have what effectively Is veto power over 
Installation of a researcn project In a particular site-so that their 
own consent can be Involved to Insure cooperation during the life of 
the research. In ether words, If prior consent would create a setting 
In which staff felt responsible for the health of the research them- 
selves, should evaluation researchers "recruit'' staff participants? 
In practice, such overt solicitation of staff support has distinct 
disadvantages. First, In the WIN context (for example), counselors 
and Job developers are not trained In social science research methods. 
They should not be expected to appreciate- let alone approve of-research 
goals and design. Feasibility research In particular (and by definition) 
entails a risk of program fallure-an aspect that regular program staff 
understandably find discomfiting. The research In Portland placed new 
demands on local WIN staff; their resistance to these demands was a 
healthy sign that they were continuing to act In their role as WIN 
CQ4ioselors-not para- researchers. Had the staff been eager for the 
voucher project, It would have suggested that their own procedures were 
more flexible or less regulatory than researchers thought, and/or that 
they were wlllln9 to step out of their roles. Yet, those roles and the 
responses of staff In those roles were among the re^search Interests In 
Portland, in general, oo-llne staff might well be made curious or Intel- 
lectually Interested by Innovations proposed In a study design, but 
given the natJre of their roles and their Informal procedural sub-systerrs 
(such as the ones discussed In the preceeding chapters), the expectation 
should perhaps be one of resistance and conflict rather than total 
cooperation, * 

Goal Conflict and Indifference 

The type of conflict that researchers experienced was not always 
verbal or argumentative, but structural. |n Portland, for example. Job 
developers who believed that clients do not know how to choose employers 
felt that the project's goal of maximum client autonomy compromised WIN's 
goal of rteximum placements. Researchers were unable to reduce this 
conflict since they had no power to add meaningful rewards (promotions or 
raises) or remove penalties (for low placements) that staff feared. 
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Sinc« th« r«f««rch itf«lf was carrUd out through a subcartract to the 
stata of Oragon, tha stata was tha logical adminl ttrat Iva unit to offar 
rawardt to participants vfho mada tha rasaarch axpeclally successful. 
For thair part, though, Oregon officials ware not Interested In allo- 
cating their scarce resources In tha form of bonuses for a project to 
tM\ch thay ware largely Indifferent. 

State officials are not required to specify their level of 
conmitMant to a federally sponsored research project before It Is Installed 
In their area. In fact, research may not fit particularly well Into 
their agandae, except colncldentally, as when an extra buogat or job 
slots are Involved, Lower level staff are caught In a triangle between 
their indifferent state supervisors and the federal govarninent, %*ho they 
see as being represented by the researchers. To soma extent, stata and 
regional adnlnl.Urators feel the same bind: caught between ''orders'* from 
Washington and their responsibilities to the other administrative unit. 

In the case of Job developers tn partlculer, the fear of penalties 
was probably a stronger force against cooperation with tha project than 
lack of Incentives to do so. Their only access to rewards (both personal 
and collective) was through high placement levels. The local supervisor 
rejected an offer from the national office to hold developers harmless 
on placements (or lack of placements) made during the life of tha 
voucher project. At laast on^ Department cf Labor official believes that 
they were actually being held harmless anyway. In any case, the matter 
was out of the hands of researchers. Furthermore, the fact that 
researchers did not seem to feel that client autonomy would necessarily 
result In a lower placement le^«l only reinforced staff belief that they 
did not sympathize wlth-*or even understand-- the problems the local 
office faced because of the research. 

Rasponsat to Conflict: Communication 

Throu^ut tha life of tha voucher project, conflict Itself was 
lass destructive than the responses to It. For exampla, during Portland 
I, tha fact that staff ware hoarding vouchers to distribute to what they 
considered to be the most promising clients (see footnote, page 13) had 
a ready solution (de-rat ioning tlie Nouchers), and was In and of Itself 
a datum on feasibility; however, at the time, the event had serious 
overtones of rebellion on the one hand and feelings of betrayal on the 



3li 



-29- 



other. Part of th« crifli mood any time rafaarch operations waro. 
disrupted In this My stemned from the feet that both staff and 
researchers did not have ecsy eccess to reeiedles. Action required 
authorization, and authorization required coMiunlcat ion, which wes 
sldtf and cumbersome. 

The lack of quick channels of communication between the 
research site and the National Office was, perhaps, a second major 
variable In the history of the voucher project. Researchers were 
restricted officially to communication through the same channels 
as other WIN operations, i.e., through the local office to the state, 
then to the region, and finally to the national office. From Its 
end» the national office could not conimunicate directly with the 
local office either; rather, those administrators, too, sent recomnen- 
datlons and directives back to the site through the region and the state. 
Inserting seven steps be* «rt question and response was frustrating 
to all participants. It also caused concrete problems. First, It 
Involved middle-level (state and regional) officials In the problems 
of research to * greater degree than they thought acceptable. Second, 
It encouraged ad hoc probienv-soi ving at the local level by researchers 
and staff themselves. Finally, it also encouraged violations of the 
channels of communication; direct telephone calls sofeetimes replaced 
properly routed memoranda. This could only execerbate feelings of 
tension In those administrators who were passed over, since it en^haslzed 
their lack of control over the research process. Part of this problem 
might be solved In the future by creating special lines of communication 
for research-related matters; these lines might not necessarily follow 
the route for regular program coflmunicatlons. 

. Research Needs vs. Service Peilvery Needs 

The fact that vouchering took place In a dlchotomous setting 
compounded the effects of the staff's InaC^iifty to adjust (at all levels, 
but particularly at the iowost level) and the difficulties of coimiunlcet Icn. 
On-line staff were being asked not only to change their roles but also. 
In effort, to perform two roles at once: researchers expected them to 
be able to do their regular WIN Jobs as If the research did not exist, 
and then to voucher, as If 'their regular jobs did not exist. In retro- 
spect, the point should be not that this expectation was excessively 
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high, but that researchers and staff alike had no means available for 
dealing with the necessary conflict that existed betvwen the two roles. 

The necessarily uneasy fit between research and Its setting 
suggests that program autonomy should not be a test of success or a goal 
for demonstration projects such as the voucher project. As the history 
of the voucher project reveals at any number of points, research 
entails special constralnts-cnost ly In the form of human relatfons-that 
regular operations do not. Thus, *>ease of fit" Is not In and of Itself 
a test of feasibility because under "real" conditions, the context for fl 
would be entirely different. For example, even counselors who did not 
find voucher', themselves objectionable or difficult to deal with, 
cofi^la:ned of the effect the project had on their ordinary routine. 
These peripheral problems would not have existed had vouchers been the 
ofJy method of operation In thist office. Furthermore, the middle level 
(state and regional) administrators did not feel that the research was 
a priority of theirs; they are especially uncomfortable with research 
and Its problems, as has already been described. 

Instead of maintaining program autonomy as the goal, perhaps 
research autonomy— as defined by the authority to maintain a stable (not 
a "pure") operation— should be. Although researchers would risk Incurrin 
massive noncooperat Ion among staff at all levels (might In fact, be unabi 
to locate a site), they would ^aln an ability to track the Implementation 
process, so that there would be something concrete to evaluate at the 
end of the experience. The voucher project took a position somewhere 
between these two extremes (project vs. program autonomy). In that It 
yielded to the priorities of regular operations selectively . 

This question of just who should be In control of research Is 
really a question of whether research is just another form of service 
delivery or whether It Is something different. In the history of the 
voucher project, service delivery was an outcome of the research, not its 
primary goal. In other words, the procedures made an effort to protect 
clients from the effects of vouchers, should they prove unacceptable 
to the training cONMunlty, for example, but the researchers did not 
ue themselves as being In the "voucher business." In fact, counselors 
sometimes accused researchers of -Mettlng clients fall" with vouchers; 
such statanerts were Indleations of the divergence between operations 
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goals and research goals. The Jurisdiction of administrators over 
both regular operations and research In the flelo helps maintain the 
Illusion that researchers shou Id be offering some kind of program, 
or that they do It badly. Special lines of communication and authorlty- 
as suggested earlier, would make researchers' special functions and needs 
more difficult to forget In dally practice. Some administrators 
believe that evaluation research should not be under the control of the 
relevant agency at al 1 i that researchers would have more freedom If 
all research were funded directly by Congress or by GAO. This sort of 
third-party research would relieve researchers and, in this case, 
counselors and job developers alike of some of the triangular relation- 
ships they found burdensome, although it would undoubtedly create some 
equally complex working relationships among agencies. 

Implementation and Evaluation 

The variables discussed in this paper should not be construed 
as signs of failure of the experiment or of the researchers (or, from 
the other side, of the staff-participants). It should not be necessary 
for the relationship of research to regular operdtions to be one of 
competition; the goals of each enterprise are, in fact, substantially 
different and divergent. While the goal of operations is, perhaps, 
uni-dimens tonal enough to fit comfortably within criteria of success 
that are purely in dollar teams, research goals are not. As we have 
seen, in the course of installing a research operation in place, 
success can be measured In effective working relationships long before 
other sorts of costs and benefits emerge. A research dollar simply 
does not buy the same sort of service that an operational dollar does; 
while this outward fact Is well recognized, the other factors Implied 
in the distinction are often not recognized or Acted upon, as we have 
seen . 

The relationship of the research and the researchers to their 
administrative environment, both In the West (Portland, Salem an(! 
Seattle) and In Washington, D.C., was the sum of its parts. Almost 
no changes could be made in any one spot without affecting the entire 
enterprise, corrective measures sometimes seemed to produce new problems, 
in this aspect, the voucher project was not unusual. The consequences 
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w«re not the dtstroctlon of th« project, or its failure; In fact, there 
Mas a fees lb I II ty test and voucher I ng did take place under some condi- 
tions. Rather, the effect of research/administrative Interaction was 
to obsure the lines between cause and effect, and to contribute a targe 
dose of aiiri>l9uity to the tasks of obtaining tnd Interpreting results. 
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SUMMARY 



The Voucher Project unfurled In three phases, each one unplanned 
at the Inception of Its predecessor. The underlying interest throughout 
was to test the feasibility of vouchers as a means for clients to purchase 
training requisite to employment. In practice, vouchers were not only 
, "blenk checks'»--wlth limits set by the researchers and the state and 
federal administrators; they were intended to be used on the basis of 
clients* decisions. The presumption was that traditional service 
delivery In the area of employment- related training was dominated by 
counseling staff. Thus, at the beginning of the research, vouchers 
represented not only a flexible, though I Lilted, amount of dollars, 
but were thought to Involve an Innovative decltlon-meklng method, as 
well. Early In the life of the project. It became clear that the line 
between voucherlng as a process end traditional decision-making methods 
was not as clear as researchers and faderal officials had presumed. 
Counselors did not perceive themselves as dominating the decision- 
making process, but only guiding It for the protection of clients, 
and they were reluctant to give up what thay saw as a necessary and 
beneficial role. They saw voucherlng In tarns of new source of 
doll s. not a new delivery system. Local and state administrators 
also tended to see the project less In terms of reseaich than In terms 
of resources. Thus, the pattern of cooperation and acconmod^tion that 
developed during the life of the project had a large element of coinci- 
dence to It. although soma staff at all levels showed Interest and good 
will In at least an abstract way. 

The process of fundi ^ the voucher project through Its phase 
of Institutional training. oi-the-Job training (OJT), and finally a 
brief open voucherlng phase, me.nt far more than juggling members on 
tally sheets, funding was an Important, structuring factor In the 
relationships between the agencies (SAU and ES) and between the agencies 
and the research, since funding determined their ability to cooperate 
substantively with research needs. As we have seen, local admlnlstra- 
tors were not autonomous In this regard, but were subject to Congress 
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•fld th« national , off ic«s of tha Dapartmants of tabor and HCW. Furthar, 
funding dae'^tnns involved malcing prior astmnptions about tha bahavlor 
of cllantt, v«nuor9 and anployart. Would ciiantt taka vouchars 
thair Mximum valu«7 Would schools or employart axpioit the time and 
dollar limits? DfsagreMaant over thata questions translated Into debate 
over exactly what a dollar would purchase when it was in the hands of a 
*1ient witK a voucher. Fineily, in the case of Port lend tl (OJT) and 
Portland Ml (open vouchering), the problems due to funding eltered 
end intensified the reietlpriship of the researchers to the Department 
of labor, since questions of allocation and timing were not matters 
that ISSR could resolve on its own. As we have el reedy seen in the 
earlier discussion in this paper, the wedge thet appaared between the 
researchers* end the agency^s view of each other*s proper function had 
a serious effect on the morale of the pMject et < whole; Salem end 
the Region were ceught in the middle, end counselors becema Increeslngly 
skeptical about the Integrity of the voucher program. 

The victim of this relationship was the reseerch: DOl was some« 
tlnas an ally of, end sometimes a detractor from, Its own objectives. 
While this ambiguous role produced difficulties for everyone involved 
Ir the project, It was doubly difficult for the program staff In 
Portland) since they felt the repercussions of flux without heving 
eccass to complete information about Its sources* The staff's ^lutlon 
as we have seen, was to ttike control by turning its collective back on 
the research. 
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A COMMENT ON SOME LARGER ISSUES 
*nn Richardson 



Tli« evants described aoova hav« significance bayond thalr Intact 
on a single project. People experienced In demonstration or evaluation 
research mMI Vecognize them, at least In form, as common problems associated 
with research In real settings. Admlnlstrstors, too, will find similarities 
with other on )y-partlal ly-successful attempts to Implement programs or 
program Innovations. 

The evaluation literature con\;alns meny examples of "the perennial 

frictions that beset applied research,' (Weiss, 1973, p. 53) and their 

consequences. Weiss has published useful sunnarles of what the e?cperlence 

of others In the past suggests are six general sources of problems deriving 

from the fact that research and practice are being carried cut In the same 

real setting. One source of friction Is that the modal personality types 

of practitioners and research people differ (or are believed to differ). 

The researcher Is likely to be a detached Individual Interested 
In Ideas and abstractions. Ke thinks In terms of generalization 
and analytical categories. ... The practitioner, on the other 

hend, Is likely to be a warm, outgoing personality, now 

conmltted to action. He finds the researcher's skepticism 
uncongenial, and he finds It difficult to warm up to him as a 
human being. (Weiss, 1972, p. 99.) 

•further, the role content of the two kinds of position Is often Inconsistent. 

Saslcally, a practitioner has to believe In what he Is doing; 
a researcher has to question |t. This difference In perspective 
creates Inevitable tensions. Whatever their Initial personal or 
value characteristics, once they go about the divergent tasks, they 
are almost bound tc see things differently. (Weiss, 1972, p. 99.) 

Along with such differences In general role content. It Is often the case 

^('t' the re'^^ctlve roles of the research and operations personnel are 

not clearly defined. 

Evaluation often requires practitioners to take on new roles. 
The new roles may not be clearly defined In advance and become 
apparent only after a series of disputes with the evaluators. . . 
Particularly frustrating are uncertainties about the authority 
struclfure; It Is often unclear who has authority to resolve the 
differences :hat arise. (Weiss, 1972, p. 100.) 
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Fourth, the goals, values and Interests of the parties to the enterprise 
are often widely divergent. 

The practitioner Is concerned with service. He sees evaluation 
as a diversion and possibly even a threat. (Weiss, 1972, p. 100.) 

[Because there Is a tendency for program staffs to see the 
evaluation effort as a potential threat to the program Itself,] 
the effects on the. study range from the annoying to the disastrous. 
[Staff at all levelsjcan refuse access to Information and to 
people; they can refuse to allow control groups to subvert them; 
their record-keeping nmy be Incomplete and faulty; they may 
manipulate the data . . . .(Weiss, 1973b, p. I80.) 

[It often appears] that the relative autonomy given to 
evaluators. . . to pursue their research was motivated less by 
respect for the integrity of research than by unsophlstlcation 
about the possible effects of evaluation. ... As this . . . 
becomes better understood, there may be more Interference with 
the planning and conduct of evaluation research. (Weiss, 1973a. 
p. SI.) 

At a slightly different level of analysis, problems are often 

created by tendencies for programs to accommodate to their surroundings, 

and to change In the process. Althougti for research purposes we require 

a relatively stable object of study, it very often is an Inaccessible 

goal In real settings. 

The program should remain stable and wel l*def Ined, so that It Is 
clear vi^at stimuli brought about the observed outcomes. But many of 
the programs [about which Weiss Interviewed research staffs] shifted, 
lurched about, and sought new directions. In these cases, before- 
after outcome data are difficult to Interpret, because the definition 
of "the program^' associated with given outcomes Is not clear. (Weiss, 
i973a, p. 51.) 

Finally, certain characteristics of the Institutional setting of 

the research can lead to less comfortable accommodation between research 

and administrative staffs. 

When an agency has a history of Internal conflict, evaluation 
mey be viewed with particular suspicion. Staff are apt to see the 
evaluators as management hatchet men. . . . Other aspects of the 
Institutional settlrg have consequences as well. Evaluator^practltloner 
relationships are affected by such aspects of the agency as the 
administrative structure, . . . supervisory practices, openness of 
communication channels, and the state of relationships with cooperating 
agencies who refer participants, receive referrals, or offer comple- 
mentary services. Where ambiguity and fragmented authority flourish 
the evaluation Is apt to suffer the strains of mlsperceptlon, con- 
flicting goals, and inadequate Support. (WeIss, 1972, p. 101.) 
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GrMnhouM^t Mt«rlalt jfftr •xnnpUt of —ch of tH«t« tourc«t of 
strtln. WhIU It would b« Inappropriate to comntnt In d«t«II on th« 
perfonallty structurts of th« Indlvldu«lf Involved at •W Uv«lt of th« 
Portland projact, It m«s cartalnly th« c«t« that th« rataarch itaff «^ra 
Incllnad to think "In tarmt of ganaral Izat'on and analytical catagorlat/' 
«nd that wa avantually bacama charactarfzad at "Eaftarn llbaralt." And, 
too, an Immadlata action orlantatlon, and protactlva concarn about tha 
>fi%\fBr9 of cllants who might ba Involvad In tha vouchar program, was 
charactarlstic of many of tha oparating itaff thara. 

DIffarancaf In rola contant oftan ratultad In avaluatlon*practIca 
conflict. BSSR's pofltlon of "lat'i try tha lyitam and saa wKat happant," 
claarly dictatad by iha faaslblllty task, was oftan acutaly discomfiting 
to tha WIN itaff, who found It difficult to think about "trying*' vouchar- 
Ing without trying to maka It work ona way or anothar. At a much Ttftar 
staga, whan tha natura of tha eSSK national offica ralatlonship bacama 
ambiguous, tha situation was avan mora dU^fassfuI ; In rha ayas of many, 
we had lost most of any lagltlmacy we aver possassad. 

Further, thara was often a lack of clear diffarentletlon of roles 
lr\ the program, though we ha«^ triad to make differences as large as 
possible by having no direct Involvement In the actual administration 
of the voucher program. However, the fsct that tha WIN staff often 
had to resort to BSSI^ staff's Interpretetion of the operating procedures 
(rather than, say, higher-level WIN steff) m«s Just one Instance of the 
blurring of the respective responslbl 1 1 ties of the two groups. 

The divergence of SSSI^ and local WIN Interests was evident et 
a number of points: In uncertainty about whether (end when) research 
objectives should prevail over, or give wey to, oparating demands; In 
the persistence with wh?ch some WIN steff selected the clients whom 
they thought would do best In the voucher program; In the delaying effects 
of the exheustlon or freezing of regular program operating funds; and 
the necessity for resaarch purposes thet the voucher program be el lowed 
to fall If It In fact was not a feeslble one, for the WIN program, 
a necessity which struck the WIN staff as Irresponsible (or worse). 

The InstltuMonal setting of the voucher effort, too, had Its 
effect. Certainly, the regular WIN program had been buffeted by repeated 
and frequent chenges In operating procedures and objectives well before 
0SSR arrived there. The change from WIN I to WIN H, with Its substan- 
tially greater emphasis on placameiit than on training, was especially 
troublesome to some staff (but by no means all). An.^ the problems which 
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d«velop«d fater in trying to articulatt the participation of the SAU 
were simply new manifestations of old t)robl«ns In coordination. Perhaps, 
however, the best suMMary of the Institutional setting Into which the 
voucher program was placed is that cited by Greenhouse: the presence 
of the project ••was announced in a memo— as one counselor said later, 
Mike the thousands of others we get around here about new rules."* 

Some of the Suggested Remedies 

The question Is, of course, how those who carry out (and, for 

that matter, those who sponsor) evaluation research can best cope with 

these several kinds of threats to the validity of their work. Again, 

Weiss has summarized much of what is known about "what to do," although 

she cautions that: 

Very little empirical research has been done on arrangements 
•nd methods that lessen tension In applied research projects. w« 
therefore have to depend for guidance on the ^'received wisdom,*' the 
generally accepted lessons of experience. (Weiss, 1972, p. 10^.) 

Assent to housing the research effort Is important: *'lt is essential to 

involve project administrators and managers In planning an evaluation.** 

(Weiss, 1972. p. m.) More specifically, Weiss argues that It is to the 

advantage of all parties to Involve practitioners in the evaluation. 

They gain understanding of what evaluation is all about. . . 

This knowledge dispels some of the sense of threat. . . and 
some of the suspicion generated by the presence of alien characters 
asking questions. . . . They have information and ideas to contri* 
bute. . . . Further, they are more likely to be cooperative about 
new procedures. . . when they see the sense of the requests. (Weiss, 
1972. p. 105.) 

Like all general recommendations, however, this one Sas Its shortcomings. 

Weiss quotes a member of a research staff attempting to evaluate a 

community health program: 

(The staff] resisted filling out records. We couldn*t devise 
a reward system to motivate them. . . . They were Involved in 
developing the record system from the very beginning, so it*s 
not a matter of the system being imposed. (Weiss, 1973a, p. 52.) 

Greenhouse also has some pertinent reservations about this reconmendat i on 

on page 

As another threat- reduc Ing strategy. Wei ss suggests emphasis on 
the "theoretical" nature of the research, Its focus on more general 
I ssues . 
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Almost every evaluation it out to discover more than whether 
this particular program works In this particular time and place 
with this staff and these participants, . , . There Is some kind 
of theory implicit In almost every program. If the evaluator can 
draw It to the surface and malte It the central focus of the evalua- 
tlon effort, he Is on the way to alleviating the very real uneasiness 
that practitioners feel about being Judged end having their per- 
formances crlMcally rated. (Weiss, 1972, p. 106,) 

Another possibility, the reduction of mystery and the fear of secrecy by 
regular provision to administrators of developing Information, is subject 
to the weakness that It may very well result In modification of the 
program itself, and thereby Jeopardize the original resft^rch objectives. 

A suggestion which has come out of others' experiences VI th 
action research Is to reduce conflict by seeing to It (by training or 
by recruitment) that research and operating staffs are more systematically 
trained In each others' concerns and objectives. 

Thus researchers with program experience would be more aware 
of, and responsive to, practitioners' perceptions and needs. One 
project [the research staff members of which were Interviewed by 
Welssl provided a clue to the utility of the prescription. The 
research director was a former [practitioner as were the senior 
researcher and a third staff member], while they were personally 
respected and their research project was accepted by the service 
staffs, they were no longer perceived as colleagues. According 
tc -ne Informant, 'They have leaped the fence and gone over to the 
otr^r side.' (Welss, 1973a, p. 53.) 

Greenhouse also discusses briefly a series of partial remedies 
for the kinds of difficulties we experienced In Portland and which others 
have had elsewhere. These include manipulation of the reward system, 
repeated clarification of authority relationships, specialized coimunl- 
catlons systems (especial ly when the research Involves several levels of 
sponsoring and host bureaucracies), dear statement of research and 
operational go«l priorities, and, perhaps, a general dissociation of 
research sponsorship from the organl2atlon(f) who have— or seem to have— 
a vested .Interest In research ar.d program outcomes.' 

Suggestions for Another Approach 

I win argue that the purposes of the sponsors and performers 
of evaliatlon research alike would be substantially served by the adoption 
of a slightly different approach, the development of a two-level evalua- 
tion research strategy. It Involves the systematic examination of the 
degree of Implementation of the program in question as a routine first 

Vhls suggestion has also been made by Leonard Goodwin (1973) and 
by Laurence E. Lynn, Jr. (1973). 
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step, prior to the study of the progrem outcomes of interest* It Is 
clear that evaluation of the latter without systematic ( and measured) 
Information on th€ former entails the risk of b^ing Incorrect to a degree 
which no performer or consumer of the research wishes to tolerate. 
Hargrove has put the suggestion In these terms: 

It Is Important to' make a distinction between Implementing a 
program and evaluating Its effectiveness In achieving intended goals. 
A policy and* program may be completely In error In the analysis of 
the problem at hand and may therefore fall to be effective even though 
beautifully Implemented. We cannot tell If a program Is properly 
targeted for a given problem If It Is badly Implemented, however, and 
this seems to be the more cofrmon case. In fact, the literature Is 
rife with negative examples. . • . . . .In general, we find it 
difficult to say whether the limitations of the p/ogram reside in fts 
Initial diagnosis of the problem or In the actual Impiemeetat Ion. 
In either case, the study of Implamentat Ion can be differentiated 
from that of program avaluatlon in its Interest In how the program 
Is carried out rather than in how well It works. (Hargrove, 1976, 

p. n.) 

It might be added that a program may also spuriously appear to work 

because of improper or incomplete Implementation, as was the danger in 

the case of the Intercession of Portland WIN Job Developers in the 
2 

vouchered OJT process. 

The remainder of this discussion will be focussed on description 
of some of the elements which might go into a systematic approach to 
Inplementat ion analysis, and on suggestions for some of the ways in 
which a two- level approach to evaluation research might increase the 
clarity, comprehensiveness, and sophistication of the research results. 

As a starting point, there are several facts of program research 
life which should be recognized as given in some degree in almost any 
real se'«.tln9, and which no amount of tinkering or ad hoc problem solving 
will eliminate to our full satisfaction.^ 



*■ For more detail on this, and on Its effects on the evaluation 
data, see Richardson (1977). 

^ The suggestions for coping with situation strains described above 
are all useful in one degree or another and in a variety of specific 
situations, and should not be ov 'looked. But by turning the technical 
question around a little, and putting the emphasis on ways in which 
systematically to take these kinds of disruptive events into account and 
in analysis of evaluation data, we may be on firmer ground than Is 
presently generally the case. 



PIrst, organ Hat Ions wTH adopt Innovations to fit as comfortably 
•s possible Mlth axlstlng and •mtrglng Intarasts which ara Ilkaly to ba 
external to the program being tested. Progrem staffs will continue to 
be concerned abo.t the performance of their regular tasks and will 
continue to exercise considerable Ingenuity In co-opting and modifying 
nevtf program elements. 

Second, events external to the experiment as such wl I i occur. 
Not all of the?n will have equal Impact on all aspects of the program 
under study, nor or all clients of the progrem (especially If the program 
operates over some period of time). In the Portland case, the deepening 
of the recession at the tlm^ the vouchered OJT program was In operation 
probably had roughly the same Impact (In the aggregate) on the voucher 
clientele. But the temporary freeze of SAU funds during that seme pro- 
gram had differential effects, end created at least two groups of clients— 
those who chosa vouchered OJT (before the freeze) and those who chose 
VQfl. and whose only option therefore was vouchered OJT (In the case of 
those clients who required child care or medical services). Similarly, 
the temporary transfer of regular OJT funds to other WIN programs In 
Oregon generated two different groups of cl lents—those who chose vouchered 
OJT and those who opted for OJT as such (and thus necessarily 
vouchered OJT), It Is not difficult to take these kinds of differential 
effects of external events upon the program Into account enplrlcally, so 
long as there are not so many of them that the case base for analysts 
becomes unmanageably small* But the Important thing Is to know Just what 
It Is that happened and when, In order that appropriate empirical Indi- 
cates can be devised and analytic measures taken. 

Further, programs will usually drift, Ndt only will they "lurch 
about" but. In the case of complex programs, will do so at different 
rates. What Is more. It Is usually In social programs that what may be 
described as "a" treatment condition Is In fact a whole series of treat- 
ments, and that they wlll'drlft In different ways and In response to 
different forces. In the voucher case, for example, "the" treatment 
Involved several dimensions: client autonomy In several decisions, 
negotiation with potential trainers, s«l f-assessment counseling, direct 
access to labor market Informetlon, an aMowance to defray some of the 
expenses associated with the search for a training position, and 




provision of a variety of support sarvlcas during the training search. 
It was our experience that the several aspects of '*the'' voucher treatment 
condition changed and drifted at different times and rates of speed (the 
first to go was self-assessment counsel lng» the last the training 
search al lOMance) . 

Next, as many have noted, some degree of conflict of research 
and operational goals probably "Inevitable," to the extent that It 
Is endemic to the structure of research In operational settings (and 
there Is reason to believe that most It). But, as Greenhouse has 
argued, goal disparities of this sort are probably also desirable: If 
ongoing program goals are fully and consistently mesr.ed with the program 
Innovation under study, the "Innov^tioti'' probably Is not that at all 
(as was nearly the cass with vouchered Institutional training In Portland). 
Or, In a different sort o^ case. If there were no strain between research 
and operdtlonal objectives, we might have grounds to suspect that the 
research (or the research staff) had been co-opted,^ a situation which 
also works to the detriment of the research by minimizing the differences 
between the "old*' and the "new." 

In light of these and other factors, it seems reasonable to 
argue that the Implementation of any program Is most accurately concep* 
tuallzed as a variable , one which Is susceptible to quantification and 
manipulation Just as any other quantitative variable Is. I suggest that 
we make the (conservative) assumption that Implementation will vary from 
one administrator, one time, one place, and with respect to one program 
aspect or another, and that these variations will have both direct and 
indirect effects on the hypothesized program outcomes. To the extent 
that this Is a valid assumption. It also means that %^at we know about 
program effects-*and what we can know- Is a partial function of these 
kinds of factors, if we were to develop measures of the degree of Imple- 
mentation, It would be possible to take account of these variations (Just 



Everhart (1977) reports his experiences In trying to construct 
and maintain a certain amount of conflict In order both to establish 
and legitimate hU work In a school. Generally speaking, anthropologists 
are more sensitive to the need to maintain constructive distance than 
are practitioners In other fields. 
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•s we do with other verleble*) In the enelysl* of program outcomes,^ 
end Co Increase the explanatory power of our analyses. 

The analysis of data on the Portland voucher project provides 
several examples of the additional understanding which can ba achieved 
when It Is possible to Include Implementation variables In the analysis 
of outcofjws. One of them has to do with variations In the Implementation 
of the vouchered OJT program so far as the matter of negotiations with 
OJT employers was concerned. As Greenhouse has described, this was an 
Issue of special sensitivity to the Job Oavalopars In the program, and 
there were several Instances In which WIN staff found the tarnptatlon 
to Intercede I rreslstebia, with the result that there ware really three 
groups of ''successful" OJT voucher recipients; those who found a training 
situation entirely on their own (which was the variable of Interest to 
the research), those on whose behalf WIN staff Intervened., and a residual 
group whose training arrangements were "mixed." Ordinarily, one might 
analyze these "comml ttment-of- voucher" data In terms of varying coninlt- 
ment rates for different demographic and program-exposure groups, and 
compare the training arrangements of the voucher conmltters with those 
of their "regular WIN" OJT counterparts. Wa did, In fact, do this, with 
the following (selected) results. 



Vouchered Regular 

OJT OJT 

Coninltment Rate of: 

Hen 18 ONA 

Handatory Women 19 qnA 

Volunteer women 23 ONA 

Average Cost of Training $2,239 $1,5^5 

Rate of Training Completion 38 33 



The extent to which this Is not routinely dona Is especially deer 
when It comas to so-called post hoc evaluation, whan one Is commissioned to 
evaluate program outcomes after they have occurred, A representative exarrple 
It that of the study conducted by BSSR of factors In WIN program experience 
whIcN might explain what seamed to be unusually high program dropout rates 
of young participants (under 22), a survey of former participants in four- 
teen cities. One of the most consistently strong effects In the aggregate 
was that of site, or city. For example, the proportion of young clients 
who were put Into OJT arrangements ranged from none In one site to kO 
percent In another (Richardson and Ounning, 1975). Clearly, the WIN program 
was being Inrplemented quite differently In different cities, and general 
statements about the outcomes of WIN participation for "young clients" 
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Quite different results were obtained, however, when we took einpl r leal ly 
Into account known variations in the degree of staff Intercessions In 
the OJT arrangements (the degree of Improper Implementation).^ 





Apparent Voucher 
Commitment Rate 


Real Voucher 
Commi tment Rate 
(ei ient-arranged OJTS) 


Men 


. 18 


11 




Mandatory Women 


19 


7 




Volunteer Women 


23 


5 






C 1 len t-ar ranged 
Vouchered OJT 


WIN-arranged 
"Vouchered" OJT 


Regular 
OJT 


Average cost of 
training 


$2,18^ 


$1,710 


$1.5itS 


Rate of training 
cofflplet Ion 


^♦3% 


36% 


33% 



The methodology of the study of implementation Is not so well 
developed among performers of evaluation research as is that of the 
study or program effects. This is probably due less to lack of Infor- 
mation about sou^'ces of slippage in real research settings than to r 
failure to systematize that knowledge into a generally-applicable 
analytic model for use In studying variations in Implementation. Some 
of the general provisions a such a systemat i zat ion might Include the 
fol lowing : 

which did not take these variations Into account would be misleading. 
In this particular case (and a few others—see Goodwin, 1977), It was 
possible to make statistical adjustments for these differences in Imple- 
mentation. In most other cases, the evaluator has not the access to 
what Weiss has ca'led "the program-as^operated'* which would be required 
In order to make adjustments of this sort. 

^hlle this Is routinely done In laboratory- type experimentation, 
It Is seldom systematically a feature of designs for research In real 
settings. 




I. Dtflnltlon of th« •Imntt of th« program In question which 
ar« to b« Implmnud, Among th« most Imporunt atpactt of tnls 
defining activity should ba tha dltattambl Ing of complax traatmants 
such that. tha Implamantatlon of aach componant can ba tubjactad to maasura- 
mant iaparataly. It It usually not sufflclant In tha avaluatlon '•asaarch 
par sa to astats whathar and to what axtant ''tha" traatny^nt has Its 
hypothaslzad affacts. This It particularly tha casa whan ••tha" traatmant 
Is In fact a Urlas of addltlva and Intaractlva traatmants which ara 
not mtroducad sarlatum or In savaral sattlngs*^ It Is aqually 
Important to disaggragata traatmants for purposas of tha study of 
Implamantatlon. 

DIffarant alamants of tha ovarall program can and should ba givan 
soma rating of thair Importanca to tha program and to tha avantual 
avaluatlon. Whlla tha fallura to Implamant cartain faaturas of tha program 
may ba fatal to tha antira antarprlsa Impropar or Incomplata Implamantatlon 
of othars will, parhaps, not ba so crucial and can ba gIvan less walght. 
In tha Portland projact, thara was a fairly claar distinction batwaan 
provisions which wara assantlal to tha hypothatlcal undarplnnlngs of 
tha vouchar notion, and othars which wara mora In tha natura of anabling 
or support faaturas of tha systam. For axampla, withholding from vouchar 
cllants of daclslon-making autonomy, or Intarcasslon by WIN staff In 
negotiations with potantial tratnars, wara consldarad to ba sarlous Impla- 
mantatlon problams, bacausa thasa aspacts of tha program went to tha 
haart of tha Idaas which wara baing tastad. Hypothat leal iy stated, 
the basic notions In question ware these: 

Putting decision-making, negotiating and purchasing power Into 
the hands of the cons>mrs of services will Increase the likelihood 
that Individual client need will be met adaquotely. 

Clients. • , win make consunptlon decisions which are at least 
as good as those made by program agency personnel. 

Allowing a client to choose a supplier In terms of his own needs, 
and thereby not limiting his potential choices to those made avail- 
able by an agency, broadens his options for both a range of services 
and a variety of suppliers* 

Giving the client the opportunity to participate meaningfully In 
decisions about his life will Increase his skill In dealing with a 
variety of Institutions, and will enhance his self-esteem, his sense 
of personal efficacy, and his comnltmant to the accompUshmant of 
his goals. (Richardson and Sharp, 197^, p. 2i) 
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On the other hand, certain others parts of the system seemed to 
be less centrally relevant. Self*assessment counseling and direct access 
to labor market Information were two enabling provisions: they were 
designed to transfer from staff to clients basic Information to be used 
in the decision-making process. The fact that they were utilized at the 
client's discretion may Indicate their relative centrallty to the program. 
Still other features of the program— iuch as the training search allowance 
and the provisions for child care, car repair, dental attention, end 
street (vs. working) clothes during the OjT search^ might best be 
characterized as peripheral services; the failure to Ihiplement these 
aspects of the voucher program would probably be less c-^clel. 

In practice, It would probably be useful to focus on the fact 
that "there Is soma kind of theory [aC teast] Implicit In almost every 
program," (Weiss, 1972, p. 106), and to make decisions on which will 
be considered crucial implementation dimensions, and which not-so-cruclal 

Q 

from the standpoint of correcting the evaluation data. 

Thus, we left the vouchered OJT experience with the Impression 
that the program had never really been tested, because there were 
observable attempts to subvert the cl lent-as-negotiator aspect of the 
system. (At the same time, provisions for training search money and 
for child and meoical care were still intact.) Clearly, some aspects 
of the program were more important than others. 

In general, then, the greatest weight in Implementation research 
should go to those provisions which are cperatlonal representations of 
the centra) theoretical concerns of the program In question. The 
assignment of weights to more p ripheral aspects of the program is 
probably In some degree a matter of guesswork, and may oe subject to 
surprising re-assessment In some cases. It Is an important guard, though, 



Normally, these are provided by WIN only after training begins. 

It Is easy to be fooled In these decisions, however. For example, 
quite iate In the OJT phase of the Portland project, we discovered that 
one Public Welfare Division worker In one of the branch offices had learned 
that a voucher client was receiving the training search allowance, and 
counted It as new Income. Th^s.had the direct effect of raising the 
price of food stamps for the client In question. Fortunately for the 
reseerch, this happened lete enough that It did not affect the voucher 
acceptance rate among clients of that branch office (and did not necessitate 
negotiations with another agency). 
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•galnst too-hasty Judgments that there fs nothing to evaluate (sinca there 
has been '»no" Implementation) on the one hand, and on the other, over- 
optimistic assessments of the possibilities for follow-on evaluation of 
outcomes. 

2. Development of valid Indicators of the degree of Inplemen- 
tatlon of each program component, at all relevant levels of the program 
administration. In the Portland case, this would have Included several 
administrative- level' the Intake staff who had control of Information 
on the existence of the voucher option; the counselors and Job developers 
who could (and did) grant or withhold autonomy In decision-making and 
negotiations with OJT employers; local office administrators who -ould 
(and did) Intercede In negotiations with training Institutions; state ana 

regional administrators who were to pass on training, proposals which 

9 

would cost more than certain amounts. 

3. Careful assessment of what "ordinary Interests" may be at 
stake In Implementation of the program, so that one may be sensitive 
to the points around which controversy or strain (and possible lack of 
Implementation) are likely to develop. Again, the risk the voucherad 
OJT orogram posed for Job developers In terms of deterioration of 
relationships with local OJT employers provides a useful Illustration. 
Job developers who In the previous (pre-voucher) year had negotiated a 
relatively large number of OJT contracts were considered to be those at 
"high" risk In the voucher system. This measure was helpful In under- 
standing some of the variations In the extant to which voucher clients 
were (by their otwn reports) allowed decision-making autonomy in three 
areas: whether to undertake vodchared OJT, what occupation to train 
for, and which employers to approach In seeking a training position. 
Aside from the deviation from expectation of two of the low-rlsk^ob 



For example, a serious threat to Implementation In the voucherad 
OJT program arose when a state administrator who was asked for approval 
of a "high-cost" OJT arrangement rejected It because the client In question 
had been a partlclp,int the year before In the voucherad Institutional 
training program, but had not ^ne to work afterward. The reason for the 
rejection (which was not an "official" ground for denying approval) was 
communicated to the client's team, which then took the position that It 
would no longer permit any of .Its clients the voucher option If that was 
to be a criterion for approval of vouchered training contracts. It did 
not take long to untangle the sltuatlob satisfactorily, bu*" It might not 
have been so easy a solution in oth«r circumstances. 
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d«v«lop«rs (th*y allowed autonomtr Utt oft*n •v^n thtn tht high-rttk 
fm% dU), th« rltk-autonoiiy r«lattonshtp was fairly ciMr, thou9h 
not •(facially strong.'^ * 



NIgh-rltk 
Taani 



Low risk Taant 
(Cxcludf Tmo 
Davlani, Taaait 



fropor*^on of ClUnts Kaporting 
AutoiHwy In: 

Undartaking Vcjcharad OJT 
Training Occupation 
Training Eoiployar 
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Th« ability to Idantify probabla points of ttrain and tha p«opla for 

oia thay ara a problam (and for Mhom thay ara net), can ylald valuaola 
Information on program Implanantatlon. it will not ba possibia to 
jntlclpata aach ralavant point of strain at tha outsat of tha study, of 
coursa; na^ ort^% ara likaly to an^rga through tima, and soma may turn 
out not to :«rasant an Inplamantatlon problaM at all In prattica. A 
wal l-dasignad Kfntel for assassmant of Inplamantatlon procassas will alloM 
for ccn.lnual davalopmant of naw maasuras as tha naad arlsas. 

^. Davalopmant of a robust data-gatharing stratagy, ona which 
will ba rai.tlvaly invulnarabia to da 1 1 barata attampts to dlsguUa 
variations In In^ laiMntatlon. Tha nagatlva axtraaias of this aspact of 
tha inpl^Mwntat Ion assassmant modal ara aasy.to Idantlfy. It would 
b« foolish, for axampla. to raly antlraly on official program raccrds-- 
tlyy far mo'-a oftan raflact cantral offica obJactJvas than thay do local 
off lea t^aalitlas. Nor would ona ba llkaly to gathar usaful tystamatic 
Information from an organizational suparior about tha bahavlor of lowar 
ataff (since s/ha may not in fact hava valid infoncation) . just as wa 
ordinarily prafar tc gathar avaluatlon-typa data from program participants 

'^Cla^rly, this cruda maasura of risk, which was davolopmd on an 
ad hoc basis, Is not fully satisfactory, and would raquira consld«rabla 
raflnamant If It wara to ba of ganar 1 uta In tha '^lysls of tha data. 
Th«ta findings ara raporvad hara, ^wava , for pu las of illustration 
of soma of tha pr ^sibi I Itias. 



themsalves rathtr than from othars or from rtcords, io mouM bt In a 
stron9«r pofition to assist tha dagraa of Impl^mtntatlon If tha data 
wara to coma diractly from tha ••cllantt" of tha Innovation, tha day-to- 
day administrators of tha program (at all lavals). 

Thara ara savaral standard data-gctharlng approachas which may 
ba sultabia for thase purposas, though aach offars waaknassas as wall as 
strengths. Self-admlnlstarad quastionnal ras ara aasy to standardize 
anJ an Inaxpanslva way to collect data but vulnerable to attempts to 
dissemble. The same Is true In lerge part of faca-to-face Interview * 
approaches. Interviews with program clients can yield useful data and 
ara probably not as vulnerable to deliberate misreporting, but they are 
at the same time subject to Inaccuracies of respondent recall. 

Occasional drop-In visits to the research site offer the 
potential for a more detailed and realistic view of the actual Implemen- 
tation process as It works In practice, but entails the risk that 
observations are not made at representative times during the program, 
or 'that operations are tal lored especial ly for the observation perlodj^ 
Continued on-site monitoring of implementation can markedly increase the 
accuracy of observation of actual events and behaviors, but It is an 
expensive approach and may create some unnecessary problems of conflict 
and accomtnodatlon, such as some of thos.- ^escribed by Greenhouse. 
Participant observation may reduce some of these problems and deepen 
insight, but may also Involve the risk of co-optatlon of the observer. 

The particular data-gathering strategy (or cor,*inatIon of approaches) 
which win yield a maximum of useful and valid data on In^lementatlon at 
a minimum cost In operating expenses and in organ izet lonel conflict 
will certainly vary with circumstencas, with the particular sensitivities 



, , " coryt^>^t of discussion of problems connected with part- 
time conmltments of evaluators to.the research operation^ Weiss shares 
this horror story: "In one project, far fewer patients entered the 
?n°L'T. ^ r •ntlclpated. Accordingly, the program staff decided 

to .dm t as patients the groups that had formerly been disignated as 
controls. Because he was not on the spot, the evaluator did not have a 

y^lli^^^ "f? ^"""''^^ 9''°^P»- even hear 

abouv the decision until over a week had passed and the move was 
practically Irrovocable. Weiss, 1973a, p^ 53 ) 



-50- 

12 

of the situation, with the length of the Implementation period, with 
the coiTplexltv of the program jnder study, and 50 forth. But It Is 
likely that, with further study, a generally-applicable "best option" 
approach, or a general ranking of the desirability of all options could 
be devised for usejn any study of progra.i Implementation. 

Clearly, there are a good many considerations to be taken into 
account in development of a coherent and general approach to Implemen- 
tation research ^hicH have not been addressed here. Hy attenpt has 
been to specify some of the beginnings of the development of a useful 
model. This discussion amounts finally to a proposal that systematic 
work along these lines be begun. 



12 

For example, certain implementation studies could very wflll 
Involve the revelation of "organizational secrets," and would call for 
approcches which eventually become relatively unobtrusive, such as 
participant observation. Others might not, and could satisfactorily be 
carried out with other approaches. 
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